feat(exec-harness): declare the benchmarked command's pid by lvaroqui · Pull Request #559 · CodSpeedHQ/codspeed

lvaroqui · 2026-10-01T14:53:30Z

Make exec-harness declare the pid of the command it benchmarks, so its own spawning cost can be left out of simulation results.

Since #531, exec-harness turns instrumentation on in its own process and spawns the command, which inherits that state across fork and exec. The dumped part therefore also holds the harness's cost of spawning the command and waiting for it. That is a fixed overhead on every benchmark, about 207k instructions per command locally.

Changes:

instrument-hooks binding: new set_executed_benchmark_for_pid(pid, uri). set_executed_benchmark keeps its behavior and delegates to it.
exec-harness: spawns the command and passes its pid, instead of calling Command::status(). In memory mode the pid only travels to the runner's FIFO, where nothing reads it for exec-harness.
Submodule bump: feat(valgrind): declare the pid a benchmark ran in instrument-hooks#32 writes desc: Benchmark pid: <pid> in the part when the pid is not the caller's. This bump also brings the thread-safe C API exports and the callgrind_toggle_collect helper.

Checked locally under the patched valgrind, with the runner's simulation flags on sh -c '/bin/true; /bin/true; :':

part: 2
desc: Spawned pid: 289126
desc: Benchmark pid: 289126
desc: Trigger: Client Request: exec_harness::true_twice

Still to do before this is ready:

Bump the valgrind pin to 0codspeed8 once feat(callgrind): add CALLGRIND_ADD_DESC client request valgrind-codspeed#43 is released. Until then the request is ignored with a warning and results keep the harness's cost.
Repoint the submodule once feat(valgrind): declare the pid a benchmark ran in instrument-hooks#32 merges.
The backend support for leaving out the declared launcher cost isn't released yet.

Closes COD-3722

codspeed · 2026-10-01T15:09:42Z

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 41 untouched benchmarks
⏩ 6 skipped benchmarks¹

_{Comparing cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation (fbb67ac) with spike/cod-3440-memtrack-musl (fd51df2)}

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

greptile-apps · 2026-10-09T08:32:10Z

Confidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge.

Summary

exec-harness now records the spawned command’s PID with its benchmark URI.

Simulation profiles name the process that ran each command.
Valgrind allows the runner executable’s file capabilities.

No actionable issues were found. lvaroqui acknowledges that backend support for removing the harness’s cost is not released yet. The description also lists pending Valgrind and submodule work; the current code already pins 0codspeed9 and the updated hooks revision.

Diagram

sequenceDiagram
    participant Harness as exec-harness
    participant Command as Benchmark command
    participant Hooks as Instrument hooks
    participant Valgrind
    Harness->>Hooks: Start benchmark
    Harness->>Command: Spawn and save PID
    Command-->>Harness: Exit status
    Harness->>Hooks: Stop benchmark
    Harness->>Hooks: Declare command PID and URI
    Hooks->>Valgrind: Add Benchmark pid description
    Hooks->>Valgrind: Dump benchmark part

_{Reviews (2) · Last reviewed commit: "feat(instrument-hooks): declare benchmar..." · Reviewed by Greptile}

musl has no `close_range` wrapper, so call the syscall directly. libbpf-sys vendors elfutils, whose `configure` requires `argp`, `obstack` and `fts`, none of which musl provides even though a libelf-only build never calls them: `.cargo/config.toml` seeds the autoconf cache to skip those checks, adds an `argp.h` stub through `CPATH`, and points Debian's `musl-gcc` back at the kernel UAPI headers. aarch64 also links libgcc for the outline-atomic helpers. The libc symbol resolution test now reads the Ubuntu libc path and only runs in CI, since a static musl test binary maps no libc of its own. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

Move each binary's logger setup, argument parsing and dispatch into a `cli::run_cli` library function that takes its argv, leaving `main` a thin wrapper. This lets the runner call either CLI in-process with the same logging. No behavior change. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

In simulation mode, exec-harness ran each target with an `LD_PRELOAD` library that toggled instrumentation from inside it. That needed a C build step and failed for executables that ignore `LD_PRELOAD`, such as static binaries. Simulation now runs targets like memory mode does, and the runner enables subprocess tracking for exec-harness runs so valgrind follows into the target. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

exec-harness was released on its own and downloaded at a pinned version the first time a run had exec targets. Run it as a hidden `codspeed exec-harness` subcommand instead, so it always matches the runner and needs no download. The exec-harness crate is no longer released. Internal subcommands now dispatch before any runner setup: they run in the benchmark's working directory, where loading a stray `codspeed.yaml` could abort the measurement. `codspeed samply` is hidden from the help like the others. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

memtrack was released on its own as `codspeed-memtrack` and installed at a pinned version by the memory executor's setup. Link it into the runner with its eBPF feature and run it as a hidden `codspeed memtrack` subcommand, so it always matches the runner and needs no download. With exec-harness already bundled, the binary installer has no users left and is removed. memtrack is no longer released. The eBPF capabilities are granted to a copy of the runner executable rather than the runner itself, so the runner never runs in glibc's secure-execution mode, which would strip `LD_*` variables from the benchmark environment. The copy lives under `~/.cache/codspeed/memtrack`, keyed by the executable's hash, and is created before any sudo run can leave a root-owned `~/.cache`. Copies of other builds unused for a day are removed. `codspeed setup memtrack-path` prints the copy's path. Because that copy gets `setcap`, the `CODSPEED_SELF_EXE` override of the re-executed binary is now test-only. `codspeed setup status` reports no separate memtrack tool any more, and shows privilege status for executors without one. Building the runner now builds memtrack's vendored libbpf, so CI installs the bpf dependencies for it, the release gets memtrack's build dependencies, and a new CI job builds the musl targets the runner ships. Closes COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

With memtrack linked into the runner, every memory run hashes the runner binary to locate its capability-holding copy and runs memtrack from it. In the debug build both are unoptimized: hashing the 376 MB binary alone takes about 5.6s per run, which took the memory shards from ~10s to ~42s. Build and set up the executor tests with the release profile; the memory shards are back to 8-10s. Refs COD-3440 Co-Authored-By: Claude <noreply@anthropic.com>

Add set_executed_benchmark_for_pid, which passes an explicit pid to instrument_hooks_set_executed_benchmark instead of the calling process' own. set_executed_benchmark keeps its behavior and delegates to it. Bump instrument-hooks, whose valgrind instrument now writes a "Benchmark pid: <pid>" desc line in the dump part when that pid is not the calling process'. This also brings thread-safe C API exports and the callgrind_toggle_collect helper. Refs COD-3722 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

GuillaumeLagrange · 2026-11-03T16:07:39Z

Merged in #531

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch 2 times, most recently from c72e991 to 3c1b378 Compare October 5, 2026 13:25

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch 2 times, most recently from 1af664e to 8fb4858 Compare October 7, 2026 09:06

lvaroqui added this pull request to stack #572 October 7, 2026 09:22

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch 7 times, most recently from 345f77b to b0533e3 Compare October 8, 2026 12:10

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch from b0533e3 to a301550 Compare October 8, 2026 13:34

lvaroqui marked this pull request as ready for review October 9, 2026 08:27

lvaroqui requested a review from not-matthias October 9, 2026 08:27

not-matthias approved these changes Oct 9, 2026

View reviewed changes

Comment thread src/executor/valgrind/setup.rs Resolved

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch from a301550 to 8369f55 Compare October 12, 2026 14:23

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch 2 times, most recently from 0993ffa to e294183 Compare October 14, 2026 13:23

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch from e294183 to 0049a9c Compare October 14, 2026 13:28

lvaroqui force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch from 0049a9c to 84e0796 Compare October 14, 2026 13:43

GuillaumeLagrange and others added 4 commits November 2, 2026 18:21

GuillaumeLagrange and others added 2 commits November 2, 2026 18:43

GuillaumeLagrange force-pushed the cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation branch from 84e0796 to fbb67ac Compare November 2, 2026 17:46

greptile-apps Bot mentioned this pull request Nov 2, 2026

feat!: ship one binary — bundle exec-harness and memtrack, drop the LD_PRELOAD hack #531

Merged

Base automatically changed from spike/cod-3440-memtrack-musl to main November 3, 2026 15:53

GuillaumeLagrange closed this Nov 3, 2026

Navigation Menu

Repository navigation

feat(exec-harness): declare the benchmarked command's pid - #559

feat(exec-harness): declare the benchmarked command's pid#559
lvaroqui wants to merge 7 commits into
mainfrom
cod-3722-ignore-process-spawning-overhead-in-exec-harness-simulation

lvaroqui commented Oct 1, 2026

Uh oh!

codspeed Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 9, 2026 •

edited

Loading

Diagram

Uh oh!

Uh oh!

GuillaumeLagrange commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lvaroqui commented Oct 1, 2026

Uh oh!

codspeed Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

greptile-apps Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

GuillaumeLagrange commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codspeed Bot commented Oct 1, 2026 •

edited

Loading

greptile-apps Bot commented Oct 9, 2026 •

edited

Loading